Skip to content

perf: purge in chunks of 8, not 100 - #5134

Closed
mennatnaga wants to merge 1 commit into
menna/p2p-close-racefrom
menna/purge-chunk-size
Closed

perf: purge in chunks of 8, not 100#5134
mennatnaga wants to merge 1 commit into
menna/p2p-close-racefrom
menna/purge-chunk-size

Conversation

@mennatnaga

@mennatnaga mennatnaga commented Aug 11, 2026

Copy link
Copy Markdown
Member

Stacked on #5133.

purgeChunkSize bounds how many documents share one purge transaction. It keeps a commit under the store's transaction size limit, which it does at 100, but it also sets the cost of the purge and that part was never measured.

Each purged document opens several iterators inside the chunk's transaction, and badger rebuilds and re-sorts the transaction's whole pending-write set on every one, so a chunk costs roughly the square of its size and total purge time is close to linear in the constant. It is the same cost mergeChunkSize = 8 was introduced to remove from the merge path.

BenchmarkPurgeByDocIDsChunkSize, added here, measures 2,739 docs/s at 8 against 713 at 100 and 365 at 200, on disk with three secondary indexes and history pruning on. A host runs 44 to 203 docs/s on the same code, so the magnitudes do not reconcile and this is a candidate to A/B rather than a proven fix. Sitting at the top of the stack means the treated build differs from what the test mesh already runs by this constant alone. If drain on a treated host stays under 100 docs/s the cost is in the per-document DAG walk, not the chunking.

@mennatnaga
mennatnaga force-pushed the menna/purge-chunk-size branch from c7d22ac to 2d208e4 Compare August 11, 2026 22:45
@mennatnaga
mennatnaga changed the base branch from shinzo-develop to menna/p2p-close-race August 11, 2026 22:46
@mennatnaga mennatnaga closed this Aug 11, 2026
@mennatnaga
mennatnaga deleted the menna/purge-chunk-size branch August 11, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant